Visuomotor extrapolation

Accurate perception of moving objects would be useful; accurate visually guided action is crucial. Visual motion across the scene influences perceived object location and the trajectory of reaching movements to objects. In this commentary, I propose that the visual system assigns the position of any object based on the predominant motion present in the scene, and that this is used to guide reaching movements to compensate for delays in visuomotor processing. Nijhawan’s article provides evidence for compensation mechanisms in visual perception and visually guided action. Most of this evidence is drawn from the flash-lag effect, where a single object moves across the retina. There are several other illusions, some of which are briefly mentioned in the target article, which might also support Nijhawan’s position (De Valois & De Valois 1991; Hess 1904; Matin et al. 1976; Nishida & Johnston 1999; Ramachandran & Anstis 1990; Regan & Beverley 1984; Snowden 1998; Whitaker et al. 1999; Whitney & Cavanagh 2000). For a review of these illusions, see Whitney (2002). The strongest support for compensation in the perceptual system (e.g., extrapolation) comes from the displacement of stationary edges by motion. For example, visual motion viewed through a static aperture causes the aperture to appear shifted in the direction of the motion (De Valois & De Valois 1991; Ramachandran & Anstis 1990; Regan & Beverley 1984); the motion aftereffect is accompanied by a concurrent shift in the apparent position of a static test stimulus (Nishida & Johnston 1999; Snowden 1998; Whitaker et al. 1999); and, static flashed objects appear shifted in the direction of nearby motion (Whitney & Cavanagh 2000). Whereas the flashlag effect may be due to differential latencies for moving and flashed objects (Ogmen et al. 2004; Purushothaman et al. 1998; Whitney & Murakami 1998), these other mislocalizations of static edges by visual motion cannot be caused by temporal mechanisms such as differential latencies (Whitney 2002). Although these mislocalizations of static edges by visual motion provide the strongest support for perceptual extrapolation (i.e., compensation for neural delays in the perceptual system), some of these illusions greatly complicate things: Several papers have shown that flashed objects appear shifted forward, in a direction consistent with any nearby visual motion, even when that motion is several degrees away from the flash. This has been called the “flash-drag” effect or the “flash-shift” effect (Durant & Johnston 2004; Eagleman & Sejnowski 2007; Watanabe et al. 2002; 2003; Whitney 2006; Whitney & Cavanagh 2000; 2003). Because the flash is not moving, and it is distantly separated from the moving object, it does not immediately make sense why the flash should appear shifted (or extrapolated) in the direction of nearby motion. This result is somewhat difficult to reconcile with the notion of compensation for moving object positions, but it is not entirely incompatible. In fact, this flash-drag effect suggests that the sort of compensation that Nijhawan describes for a single moving object extends to all objects, and may be a far more pervasive and important mechanism than simply allowing us to perceptually extrapolate a baseball or other moving object’s position. In Nijhawan’s article, the primary case that is considered is one in which a single moving object needs to be perceived or grasped. This is a relatively rare situation compared to what normally happens: usually, there is image motion across the entire retina, not just a single moving object. Normally the world is physically stationary, and it is we (our eyes, heads, or bodies) that move around; and it is our movement which generates retinal image motion. For example, when we reach to any object, we usually make an eye or head movement during or just before the reach. In this case, there is retinal motion of the scene and the target object. On account of delays in visual processing, delays in coordinate transformations, and other factors such as imperfect efference copy signals (Bridgeman 1995) – along with the fact that targets of reaching movements are coded in eye-centered coordinates (Buneo et al. 2002; Crawford et al. 2004; Henriques et al. 1998) – our visuomotor system faces a somewhat similar challenge to the one outlined by Nijhawan, but on a much grander scale. Because of these visuomotor delays, we should miss-direct nearly every reaching movement we make to virtually any object. Every time we reach toward our coffee cup, we should either hit the cup, knocking it over, or fall short of the cup – all because of sluggish visual and motor processing. How does the visuomotor system avoid these errors? In a recent series of studies, we found that the visuomotor system samples motion across the visual field and then shifts the trajectory of the hand in the direction of that motion when reaching to any object in the scene (Whitney & Goodale 2005; Whitney et al. Commentary/Nijhawan: Visual prediction 220 BEHAVIORAL AND BRAIN SCIENCES (2008) 31:2 2003; 2007). This effect was recently called the manual following response (Gomi et al. 2006; Saijo et al. 2005) and reveals an adaptive mechanism: The visuomotor system uses retinal motion to gauge movements of the eye and body (probably because it is as fast or faster than using vestibular or proprioceptive cues), and then adjusts the trajectory of the reach based on this information to improve the accuracy of goal-directed action. In support of this, when subjects were passively rotated, the presence of background retinal image motion improved the accuracy of reaching movements compared to cases in which only static information, or nothing, was visible (Whitney et al. 2003). The manual following response is conceptually similar to the “flashdrag effect” described above, and it suggests that the visual and visuomotor systems use retinal image motion (the kind generated every time we move our eyes) to update/extrapolate/shift the representations of object position (causing objects to appear shifted in position) – and this allows us to guide our hand more accurately than would otherwise be possible. This visuomotor extrapolation model has the advantage that it accounts for several psychophysical findings that are discrepant with the perceptual extrapolation model; and it also has the advantage that it explains accurate visuomotor behavior under the most common circumstances – where the world is stationary and we are moving. Compensation for time delays is better achieved in time than in space doi: 10.1017/S0140525X08004056 Myrka Zago and Francesco Lacquaniti Scientific Institute Santa Lucia, 00179 Rome, Italy; University of Rome “Tor Vergata” and Scientific Institute Santa Lucia, 00179 Rome, Italy. [email protected] [email protected] Abstract:Mechanisms of visual prediction based on spatial extrapolation work only for targets moving at constant speed, but do not easily accommodate accelerating or decelerating motion. We argue that mechanisms based on temporal extrapolation deal with both uniform and non-uniform motion. We provide behavioural examples from interception of falling objects and suggest possible neurophysiological substrates of time extrapolation.Mechanisms of visual prediction based on spatial extrapolation work only for targets moving at constant speed, but do not easily accommodate accelerating or decelerating motion. We argue that mechanisms based on temporal extrapolation deal with both uniform and non-uniform motion. We provide behavioural examples from interception of falling objects and suggest possible neurophysiological substrates of time extrapolation. Nijhawan makes a clear case for the need to compensate for delays arising from processing and transmission times. The evidence for compensation in perceptual decision and visual awareness appears somewhat controversial (Eagleman & Sejnowski 2000; Krekelberg & Lappe 2001), but the evidence for compensation for motor reactions to a rapidly changing sensory stimulus is uncontroversial. Typical visuomotor delays in ballistic interception of fast targets (such as in catching or hitting in ball games) are about 200 msec – at least an order of magnitude longer than the temporal accuracy required for interception (about +10 msec). Unless the nervous system has built-in mechanisms to compensate for such delays, the interception program would be based on obsolete visual information about target motion, and, as a consequence, the target would be badly missed. Nijhawan proposes a mechanism for neural compensation of delays that is based on a spatial extrapolation linearly related to the time delay. According to his hypothesis, visual prediction would be concerned primarily with horizontal processes, which transmit neural information between two neighbouring retinotopic sites. The speed of neural transmission and the distance between neighbouring neurons along the horizontal direction would jointly determine the amount of spatial and temporal extrapolation. Another mechanism could consist in a shift of the receptive field in response to moving stimuli. Sundberg et al. (2006) found that neurons in monkey area V4 exhibit such a shift in response to a particular type of moving stimuli. The direction of the receptive field shift was opposite to the direction of target motion, as if the cell had been recruited by a wave of activity preceding the target. Ferrera and Barborica (2006) argued that a moving target would leave a trail of refractory neurons in its wake so that spiking activity would be shifted toward the leading edge. Interestingly, mechanisms of visual prediction based on spatial extrapolation, such as those mentioned above, work only for targets moving at constant speed (uniform motion), because the spatial shifts co-vary with the time samples in a fixed manner. Most targets, however, accelerate or decelerate to a variable extent. Let us consider a very common situation – that of motion affected by Earth’s gravity, such as freefall, ballistic, pendulum, or wave motion. Although all objects are accelerated downward by gravity at the same rate, the corresponding acceleration of the retinal image is not at all constant, being inversely related to the apparent viewing distance of the object. The question then is how the central nervous system (CNS) compensates for delays in the case of accelerating or decelerating motion. Here we show that temporal extrapolation rather than spatial extrapolation can more easily do the job. Figure 1A depicts space–time plots similar to those of Figures 3 and 4 of Nijhawan, but for an object moving at constant acceleration (when the spatial variable decreases from right to left) or deceleration (when the spatial variable increases from left to right). The dashed curve depicts the physical trajectory, and the dotted curve depicts the corresponding trajectory “seen” by a neuron with a fixed visual delay. Clearly, the spatial shifts required to compensate for the visual delay (solid line segments connecting the two curves) are not constant anymore, as they were in the spatial extrapolation scheme proposed by Nijhawan. In theory, a first-order model might be used to approximate a second-order motion. One such model is provided by the tau function, tau 1⁄4 x(t)/v(t), where x(t) is the spatial position of the target and v(t) is the corresponding velocity (Lee 1976). However, it can be shown that, in case of free-fall motion from relatively short drop heights, such an approximation would imply significant temporal errors in interception (.50 msec), corresponding to the difference between the time-to-contact predicted by tau and the actual time-to-contact of the ball accelerated by gravity (Zago & Lacquaniti 2005). In fact, we know that unless taken by surprise, people can easily intercept targets descending along the vertical accelerated by gravity (Lacquaniti & Maioli 1989; Zago et al. 2004); they generally intercept Figure 1 (Zago & Lacquaniti). Figure depicts space-time plots similar to those of Figures 3 and 4 in the target article, but for an object moving at constant acceleration (when the spatial variable decreases from right to left) or deceleration (when the spatial variable increases from left to right). The dashed curve depicts the physical trajectory and the dotted curve depicts the corresponding trajectory “seen” by a neuron with a fixed visual delay. Commentary/Nijhawan: Visual prediction BEHAVIORAL AND BRAIN SCIENCES (2008) 31:2 221 a mechanism would have the necessary speed to produce the interception of accelerating objects, as has been demonstrated by Lacquaniti and colleagues. A time-extrapolation could supplement the task of prediction, and work cooperatively with space-extrapolation to yield a more robust system than each type of extrapolation alone would. Cunningham suggests that in addition to spatial compensation for delays, there should also be temporal compensation; and in addition to compensation for delays for continuous events, there should also be compensation for discrete events. There are perhaps two ways of describing temporal compensation. One is similar to whatz Zago & Lacquaniti might call time-extrapolation. This may be called absolute temporal compensation, as it affects delays between a physical event and its registration. The other is relative temporal compensation, where the nervous system actively coordinates sensory signals to compensate for temporal asynchronies between modalities (see commentary by Cutting). Relative temporal compensation is what Cunningham and colleagues have shown in their interesting experiments. Let us consider absolute temporal compensation further. First, there is a form of absolute temporal compensation, as demonstrated by experiments on finger tapping to auditory tones. Within limits, human subjects can predict repetitive discrete tones and put their finger taps just in advance of the tones (Mates et al. 1994). This is a good example of where the sensorimotor system must have information about the actual time of an external event. But can absolute temporal compensation occur for perception (as opposed to behavior such as finger taps)? Can the visual system generate the percepts of repetitive flashes, for example, simultaneously with the actual flashes? The answer would appear to be no, and this is where spatial and temporal compensations differ. The claim of spatial compensation is that it can put the percept of the moving object closer to (or even ahead of) the actual position of the moving object. In contrast, temporal compensation cannot, it seems, put the perceptual event close to the time of the actual event. I fully endorse the amendment suggested by Cunningham that: “The goal of visual prediction is to use priors acquired from both previous experience and the currently unfolding visual stimulus to create a perceived state of the world that matches, as far as possible, the actual state of the world.” R6.7. Mental extrapolation and (not “or”) visual extrapolation Kerzel & Müsseler suggest that sensorimotor prediction and mental extrapolation, as opposed to visual extrapolation, can overcome perceptual latencies. There is no doubt that sensorimotor prediction is an important, and highly flexible, contributor to successful behavioral acts. The role of mental extrapolation in the context of flashlag effect is, however, not as clear as these commentators propose. Kerzel & Müsseler claim that the missing predictive-overshoot in the flash-terminated condition opposes the visual extrapolation account, and they invoke mental extrapolation in explaining the forward-shift of fading moving objects (Maus & Nijhawan 2006). However, it is not clear how mental extrapolation escapes this very criticism. Why does mental extrapolation not lead to an overshoot in the flash-terminated condition? Mental extrapolation falls in the general category of phenomena such as mental imagery and mental rotation. Researchers have investigated the neural basis of mental imagery. One of the key findings is that mental imagery tasks engage the primary visual cortex (Kosslyn & Sussman 1994). In addition, Kosslyn and colleagues have found a number of similarities between mental imagery and visual perception, such as the topographic nature of both representations (Kosslyn et al. 1995). Thus, the existence of mental extrapolation would predict the existence of visual extrapolation. Kerzel & Müsseler’s proposal that mental extrapolation exists but visual extrapolation does not is unparsimonious. I claim that the task of mental extrapolation is not to solve the problem of neural conduction delays, but rather, it is to determine when a moving object, occluded by another object, will reappear (Wexler & Klam 2001). In the case of continuous sensory input from a moving object, the task of mental extrapolation is to determine the object’s future position. The task of visual extrapolation is to use sensory input to determine the object’s current position (after compensating for visual conduction delays). In past studies, Kerzel and colleagues have used either probe stimuli presented after a retention interval, or pointing movements, and so in effect asked for the remembered final position of the moving target. In flashlag experiments, or in the task used by Maus and Nijhawan (2006), participants make an online perceptual judgment comparing the position of the moving target to a flash or to a static probe. Although obviously the observer’s response is given after the visual offset, the judgment is based on simultaneously visible stimuli. It is likely that the two experimental methods differentially engage mental and perceptual extrapolation. In this context, it is interesting to note that the forward-shift effect of the fading moving object observed by Maus and Nijhawan (2006) is 175 msec, which is a much larger shift than the typical flash-lag effect of 80 msec. It is possible that this is a cumulative effect of both visual and mental extrapolation.